{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NCF Recommender with Explict Feedback"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we demostrate how to build a neural network recommendation system, Neural Collaborative Filtering(NCF) with explict feedback. We use Recommender API in Analytics Zoo to build a model, and use optimizer of BigDL to train the model. \n",
    "\n",
    "The system ([Recommendation systems: Principles, methods and evaluation](http://www.sciencedirect.com/science/article/pii/S1110866515000341)) normally prompts the user through the system interface to provide ratings for items in order to construct and improve the model. The accuracy of recommendation depends on the quantity of ratings provided by the user.  \n",
    "\n",
    "NCF([He, 2015](https://www.comp.nus.edu.sg/~xiangnan/papers/ncf.pdf)) leverages a multi-layer perceptrons to learn the user–item interaction function, at the mean time, NCF can express and generalize matrix factorization under its framework. includeMF(Boolean) is provided for users to build a NCF with or without matrix factorization. \n",
    "\n",
    "Data: \n",
    "* The dataset we used is movielens-1M ([link](https://grouplens.org/datasets/movielens/1m/)), which contains 1 million ratings from 6000 users on 4000 movies.  There're 5 levels of rating. We will try classify each (user,movie) pair into 5 classes and evaluate the effect of algortithms using Mean Absolute Error.  \n",
    "  \n",
    "References: \n",
    "* A Keras implementation of Movie Recommendation([notebook](https://github.com/ririw/ririw.github.io/blob/master/assets/Recommending%20movies.ipynb)) from the [blog](http://blog.richardweiss.org/2016/09/25/movie-embeddings.html).\n",
    "* Nerual Collaborative filtering ([He, 2015](https://www.comp.nus.edu.sg/~xiangnan/papers/ncf.pdf))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Intialization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* import necessary libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from zoo.models.recommendation import UserItemFeature\n",
    "from zoo.models.recommendation import NeuralCF\n",
    "from zoo.common.nncontext import init_nncontext\n",
    "import matplotlib\n",
    "from sklearn import metrics\n",
    "from operator import itemgetter\n",
    "from bigdl.nn.criterion import *\n",
    "from bigdl.optim.optimizer import *\n",
    "from bigdl.dataset import movielens\n",
    "from bigdl.util.common import *\n",
    "\n",
    "matplotlib.use('agg')\n",
    "import matplotlib.pyplot as plt\n",
    "%pylab inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Initilaize NN context, it will get a SparkContext with optimized configuration for BigDL performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "sc = init_nncontext(\"NCF Example\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Download and read movielens 1M data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T08:13:08.071512Z",
     "start_time": "2017-08-03T08:13:04.077308Z"
    }
   },
   "outputs": [],
   "source": [
    "movielens_data = movielens.get_id_ratings(\"/tmp/movielens/\")\t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Understand the data. Each record is in format of (userid, movieid, rating_score). UserIDs range between 1 and 6040. MovieIDs range between 1 and 3952. Ratings are made on a 5-star scale (whole-star ratings only). Counts of users and movies are recorded for later use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T08:13:08.082461Z",
     "start_time": "2017-08-03T08:13:08.075167Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1000209, 3)\n",
      "1 6040 1 3952 [1 2 3 4 5]\n"
     ]
    }
   ],
   "source": [
    "min_user_id = np.min(movielens_data[:,0])\n",
    "max_user_id = np.max(movielens_data[:,0])\n",
    "min_movie_id = np.min(movielens_data[:,1])\n",
    "max_movie_id = np.max(movielens_data[:,1])\n",
    "rating_labels= np.unique(movielens_data[:,2])\n",
    "\n",
    "print(movielens_data.shape)\n",
    "print(min_user_id, max_user_id, min_movie_id, max_movie_id, rating_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T03:45:25.657796Z",
     "start_time": "2017-08-03T03:45:25.650852Z"
    }
   },
   "source": [
    "* Transform original data into RDD of sample. \n",
    "We use optimizer of BigDL directly to train the model, it requires data to be provided in format of RDD([Sample](https://bigdl-project.github.io/master/#APIGuide/Data/#sample)). A `Sample` is a BigDL data structure which can be constructed using 2 numpy arrays, `feature` and `label` respectively. The API interface is `Sample.from_ndarray(feature, label)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T03:47:57.601392Z",
     "start_time": "2017-08-03T03:47:57.591491Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<zoo.models.recommendation.recommender.UserItemFeature at 0x10e3aa250>,\n",
       " <zoo.models.recommendation.recommender.UserItemFeature at 0x10e3aa350>,\n",
       " <zoo.models.recommendation.recommender.UserItemFeature at 0x10e3aa450>]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def build_sample(user_id, item_id, rating):\n",
    "    sample = Sample.from_ndarray(np.array([user_id, item_id]), np.array([rating]))\n",
    "    return UserItemFeature(user_id, item_id, sample)\n",
    "pairFeatureRdds = sc.parallelize(movielens_data).map(lambda x: build_sample(x[0], x[1],x[2]))\n",
    "pairFeatureRdds.take(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T03:26:59.143086Z",
     "start_time": "2017-08-03T03:26:59.136374Z"
    }
   },
   "source": [
    "* Randomly split the data into train (80%) and validation (20%)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T08:13:10.225186Z",
     "start_time": "2017-08-03T08:13:10.117969Z"
    }
   },
   "outputs": [],
   "source": [
    "trainPairFeatureRdds, valPairFeatureRdds = pairFeatureRdds.randomSplit([0.8, 0.2], seed= 1)\n",
    "valPairFeatureRdds.cache()\n",
    "train_rdd= trainPairFeatureRdds.map(lambda pair_feature: pair_feature.sample)\n",
    "val_rdd= valPairFeatureRdds.map(lambda pair_feature: pair_feature.sample)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T08:13:10.541314Z",
     "start_time": "2017-08-03T08:13:10.534792Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "799923\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[Sample: features: [JTensor: storage: [   1.  661.], shape: [2], float], labels: [JTensor: storage: [ 3.], shape: [1], float],\n",
       " Sample: features: [JTensor: storage: [   1.  914.], shape: [2], float], labels: [JTensor: storage: [ 3.], shape: [1], float],\n",
       " Sample: features: [JTensor: storage: [  1.00000000e+00   3.40800000e+03], shape: [2], float], labels: [JTensor: storage: [ 4.], shape: [1], float]]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(train_rdd.count())\n",
    "train_rdd.take(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* In Analytics Zoo, it is simple to build NCF model by calling NeuralCF API. You need specify the user count, item count and class number according to your data, then add hidden layers as needed, you can also choose to include matrix factorization in the network. The model could be fed into an Optimizer of BigDL or NNClassifier of analytics-zoo. Please refer to the document for more details. In this example, we demostrate how to use optimizer of BigDL. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T09:39:03.502185Z",
     "start_time": "2017-08-03T09:39:03.477423Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "creating: createZooNeuralCF\n"
     ]
    }
   ],
   "source": [
    "ncf = NeuralCF(user_count=max_user_id, item_count=max_movie_id, class_num=5, hidden_layers=[20, 10], include_mf = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup the Optimizer\n",
    "\n",
    "* Here, we show how to use an optimizer of BigDL to train a model.\n",
    "Optimizer tries to minimize the loss of the neural net with respect to its weights/biases, over the training set. To create an Optimizer in BigDL, you want to at least specify arguments: model(a neural network model), criterion(the loss function), traing_rdd(training dataset) and batch size. Please refer to ([ProgrammingGuide](https://bigdl-project.github.io/master/#ProgrammingGuide/optimization/))and ([Optimizer](https://bigdl-project.github.io/master/#APIGuide/Optimizers/Optimizer/)) for more details to create efficient optimizers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T09:39:07.359788Z",
     "start_time": "2017-08-03T09:39:07.321748Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "creating: createClassNLLCriterion\n",
      "creating: createMaxEpoch\n",
      "creating: createAdam\n",
      "creating: createDistriOptimizer\n",
      "creating: createEveryEpoch\n",
      "creating: createMAE\n",
      "creating: createClassNLLCriterion\n",
      "creating: createLoss\n"
     ]
    }
   ],
   "source": [
    "optimizer = Optimizer(\n",
    "    model=ncf,\n",
    "    training_rdd=train_rdd,\n",
    "    criterion=ClassNLLCriterion(),\n",
    "    end_trigger=MaxEpoch(10),\n",
    "    batch_size=2800,\n",
    "    optim_method=Adam(learningrate=0.001))\n",
    "\n",
    "optimizer.set_validation(\n",
    "    batch_size=2800,\n",
    "    val_rdd=val_rdd,\n",
    "    trigger=EveryEpoch(),\n",
    "    val_method=[MAE(), Loss(ClassNLLCriterion())]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Collect logs\n",
    "\n",
    "* TrainSummary and ValidationSummary are logs which allow user to trace how indicators (e.g. learning rate, training loss, throughput, etc.) change with iterations/time in an optimization process.\n",
    "\n",
    "A function of config_summaries is defined to collect monitoring data during training and validation phases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "creating: createTrainSummary\n",
      "creating: createSeveralIteration\n",
      "creating: createValidationSummary\n",
      "saving logs to ./log/recommender-20180510-164708\n"
     ]
    }
   ],
   "source": [
    "import datetime as dt\n",
    "def config_summaries(optimizer):\n",
    "    \"\"\"\n",
    "    :param: optimizer: the optimizer created in start_train()\n",
    "    \"\"\"\n",
    "    app_name = 'recommender-' + dt.datetime.now().strftime(\"%Y%m%d-%H%M%S\")\n",
    "\n",
    "    # train summary\n",
    "    train_summary = TrainSummary(log_dir=\"./log\", app_name=app_name)\n",
    "    train_summary.set_summary_trigger(\"Loss\", SeveralIteration(1))\n",
    "    optimizer.set_train_summary(train_summary)\n",
    "\n",
    "    # validation summary\n",
    "    val_summary = ValidationSummary(log_dir=\"./log\", app_name=app_name)\n",
    "    optimizer.set_val_summary(val_summary)\n",
    "\n",
    "    print(\"saving logs to ./log/\" + app_name)\n",
    "    return train_summary, val_summary\n",
    "\n",
    "train_summary, val_summary = config_summaries(optimizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-04T02:28:52.344745Z",
     "start_time": "2017-08-04T02:28:52.274699Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<bigdl.nn.layer.Layer at 0x11eb5f7d0>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "optimizer.optimize()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prediction\n",
    "\n",
    "* Zoo models make inferences based on the given data using model.predict(val_rdd) API. A result of RDD is returned. predict_class returns the predicted label. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[5, 5, 4, 4, 4]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = ncf.predict(val_rdd)\n",
    "results.take(5)\n",
    "\n",
    "results_class = ncf.predict_class(val_rdd)\n",
    "results_class.take(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* In Analytics Zoo, Recommender has provied 3 unique APIs to predict user-item pairs and make recommendations for users or items given candidates.\n",
    "* Predict for user item pairs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "UserItemPrediction [user_id: 1, item_id: 1193, prediction: 5, probability: 0.546546313905]\n",
      "UserItemPrediction [user_id: 1, item_id: 2804, prediction: 5, probability: 0.435872619561]\n",
      "UserItemPrediction [user_id: 1, item_id: 594, prediction: 4, probability: 0.578363712874]\n",
      "UserItemPrediction [user_id: 1, item_id: 2398, prediction: 4, probability: 0.537707021547]\n",
      "UserItemPrediction [user_id: 1, item_id: 1097, prediction: 4, probability: 0.53507514119]\n"
     ]
    }
   ],
   "source": [
    "userItemPairPrediction = ncf.predict_user_item_pair(valPairFeatureRdds)\n",
    "for result in userItemPairPrediction.take(5): print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Recommend 3 items for each user given candidates in the feature RDDs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "UserItemPrediction [user_id: 4904, item_id: 318, prediction: 5, probability: 0.905604533926]\n",
      "UserItemPrediction [user_id: 4904, item_id: 2019, prediction: 5, probability: 0.905125334014]\n",
      "UserItemPrediction [user_id: 4904, item_id: 912, prediction: 5, probability: 0.869075679543]\n",
      "UserItemPrediction [user_id: 3456, item_id: 1396, prediction: 5, probability: 0.733785094762]\n",
      "UserItemPrediction [user_id: 3456, item_id: 1374, prediction: 5, probability: 0.724252013736]\n"
     ]
    }
   ],
   "source": [
    "userRecs = ncf.recommend_for_user(valPairFeatureRdds, 3)\n",
    "for result in userRecs.take(5): print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Recommend 3 users for each item given candidates in the feature RDDs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "UserItemPrediction [user_id: 4298, item_id: 3456, prediction: 5, probability: 0.515066601913]\n",
      "UserItemPrediction [user_id: 195, item_id: 3456, prediction: 5, probability: 0.483889365973]\n",
      "UserItemPrediction [user_id: 1926, item_id: 3456, prediction: 5, probability: 0.461998483546]\n",
      "UserItemPrediction [user_id: 1271, item_id: 1080, prediction: 5, probability: 0.754923471636]\n",
      "UserItemPrediction [user_id: 2447, item_id: 1080, prediction: 5, probability: 0.709711013416]\n"
     ]
    }
   ],
   "source": [
    "itemRecs = ncf.recommend_for_item(valPairFeatureRdds, 3)\n",
    "for result in itemRecs.take(5): print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Plot the train and validation loss curves"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T09:50:45.229520Z",
     "start_time": "2017-08-03T09:50:44.643083Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd4VFX6wPHvmZIMaQRC6IHQexFCrwIqEMWCBcWCouiu\nXRdFUYT1p2LZtayFxY5iQeyCoKiIrIqCUgWlBkKHkEB6MnN+f0zJTDIlJJNMJnk/z8OTmXvv3Pue\nTHjvvaddpbVGCCFE3WAIdQBCCCGqjyR9IYSoQyTpCyFEHSJJXwgh6hBJ+kIIUYdI0hdCiDpEkr4Q\nQtQhkvSFEKIOkaQvhBB1iClUB27UqJFOTk4O1eGFECIsrVu37pjWOrGinw9Z0k9OTmbt2rWhOrwQ\nQoQlpVRaZT4v1TtCCFGHSNIXQog6RJK+EELUISGr0xdC1F5FRUWkp6eTn58f6lDClsVioWXLlpjN\n5qDuV5K+ECLo0tPTiY2NJTk5GaVUqMMJO1prjh8/Tnp6Om3atAnqvqV6RwgRdPn5+SQkJEjCryCl\nFAkJCVVypyRJXwhRJSThV05V/f7CLunbbJpFv+6jyGoLdShCCBF2wi7pv792H/d8uJGxz6wKdShC\niBoqMzOTF198sUKfHT9+PJmZmeXefvbs2Tz11FMVOlYohF3S35eRC8DOozkhjkQIUVP5S/rFxcV+\nP7t06VLi4+OrIqwaIeySfm6hNdQhCCFquBkzZrBz50569+7N9OnTWblyJcOGDWPChAl07doVgAsu\nuIC+ffvSrVs35s+f7/pscnIyx44dY8+ePXTp0oUbbriBbt26cfbZZ5OXl+f3uOvXr2fgwIH07NmT\nCy+8kBMnTgDw3HPP0bVrV3r27MmkSZMA+P777+nduze9e/fmjDPO4NSpU1X02/AUdl02z2gVzxs/\n2l8XW22YjGF33hKiTpnz+Rb+OHAyqPvs2jyOh87r5nP93Llz2bx5M+vXrwdg5cqV/Pbbb2zevNnV\nBfK1116jYcOG5OXl0a9fPyZOnEhCQoLHfrZv3867777Lyy+/zKWXXsqHH37IlVde6fO4V199Nf/5\nz38YMWIEs2bNYs6cOTzzzDPMnTuX3bt3ExkZ6ao6euqpp3jhhRcYMmQI2dnZWCyWyv5ayiXsMua4\n7s1cr9McVT1CCBFI//79Pfq8P/fcc/Tq1YuBAweyb98+tm/fXuYzbdq0oXfv3gD07duXPXv2+Nx/\nVlYWmZmZjBgxAoBrrrmGVavsbY89e/Zk8uTJvP3225hM9mvtIUOGcNddd/Hcc8+RmZnpWl7Vwu5K\nP8JkYPo5nXhy+Z8sXpfOvWM7hzokIYQf/q7Iq1N0dLTr9cqVK1mxYgU//fQTUVFRjBw50muf+MjI\nSNdro9EYsHrHlyVLlrBq1So+//xzHnnkETZt2sSMGTNITU1l6dKlDBkyhOXLl9O5c9Xns7C70gf7\nrR3ASyt3hjgSIURNFBsb67eOPCsriwYNGhAVFcW2bdv4+eefK33M+vXr06BBA3744QcA3nrrLUaM\nGIHNZmPfvn2ceeaZPP7442RlZZGdnc3OnTvp0aMH9957L/369WPbtm2VjqE8wu5KH2Bkxwo/P0AI\nUQckJCQwZMgQunfvzrhx40hNTfVYP3bsWObNm0eXLl3o1KkTAwcODMpx33zzTW666SZyc3Np27Yt\nr7/+OlarlSuvvJKsrCy01tx2223Ex8fz4IMP8t1332EwGOjWrRvjxo0LSgyBKK11tRyotJSUFF2Z\nh6gkz1gCwGMX9eDy/q2CFZYQIgi2bt1Kly5dQh1G2PP2e1RKrdNap1R0n2FZvQP2XjwA9320icJi\nGZ0rhBDlEbZJ/6O/DXa9/t+OYyGMRAghwkfYJn33yYiufePXEEYihBDhI2yTPsDITiUNujuPZocw\nEiGECA8Bk75S6jWl1BGl1GY/24xUSq1XSm1RSn0f3BB9e+Pa/q7X76zZW12HFUKIsFWeK/03gLG+\nViql4oEXgQla627AJcEJ7fS8unp3KA4rhBBhJWDS11qvAjL8bHIF8JHWeq9j+yNBiq1cpp/TqToP\nJ4SopWJiYk5rebgKRp1+R6CBUmqlUmqdUurqIOyz3P4+sh1REUagZNplIYQQ3gUj6ZuAvkAqcA7w\noFKqo7cNlVLTlFJrlVJrjx49GoRD23vx9E6y99kf9sR3QdmnECK8zZgxgxdeeMH13vmgk+zsbEaP\nHk2fPn3o0aMHn376abn3qbVm+vTpdO/enR49evD+++8DcPDgQYYPH07v3r3p3r07P/zwA1arlSlT\npri2ffrpp4NexooKxjQM6cBxrXUOkKOUWgX0Av4qvaHWej4wH+wjcoNwbACGtG/EjzuPB2t3Qogg\numPZHaw/tD6o++zdtDfPjH3G5/rLLruMO+64g5tvvhmARYsWsXz5ciwWCx9//DFxcXEcO3aMgQMH\nMmHChHI9j/ajjz5i/fr1bNiwgWPHjtGvXz+GDx/OO++8wznnnMPMmTOxWq3k5uayfv169u/fz+bN\n9v4vp/MkrqoWjCv9T4GhSimTUioKGABsDcJ+y23K4GTXa5stNNNKCCFqjjPOOIMjR45w4MABNmzY\nQIMGDUhKSkJrzf3330/Pnj0ZM2YM+/fv5/Dhw+Xa5+rVq7n88ssxGo00adKEESNG8Ouvv9KvXz9e\nf/11Zs+ezaZNm4iNjaVt27bs2rWLW2+9lWXLlhEXF1fFJS6/gFf6Sql3gZFAI6VUOvAQYAbQWs/T\nWm9VSi0DNgI24BWttc/unVUhOtLE1KFteHX1btrev5Q9c1MDf0gIUS38XZFXpUsuuYTFixdz6NAh\nLrvsMgAWLlzI0aNHWbduHWazmeTkZK9TKp+O4cOHs2rVKpYsWcKUKVO46667uPrqq9mwYQPLly9n\n3rx5LFq0iNdeey0Yxaq0gElfa315ObZ5EngyKBFV0KR+Sa5um4ey8mlav3qeQiOEqJkuu+wybrjh\nBo4dO8b339uHD2VlZdG4cWPMZjPfffcdaWlp5d7fsGHD+O9//8s111xDRkYGq1at4sknnyQtLY2W\nLVtyww03UFBQwG+//cb48eOJiIhg4sSJdOrUye/TtqpbWE6t7E2HJrGu11/9cYirByWHLhghRMh1\n69aNU6dO0aJFC5o1sz9xb/LkyZx33nn06NGDlJSU03poyYUXXshPP/1Er169UErxxBNP0LRpU958\n802efPJJzGYzMTExLFiwgP3793Pttddis9kng3zssceqpIwVEbZTK3vz294TXPSi/QG6Ox4ZJ8/P\nFSJEZGrl4JCplQPo3LTkav94TmEIIxFCiJqpViX9qAgTvVrWB+Dpr8v0GBVCiDqvViV9gIQY+4OM\nv9h4MMSRCFG3harquLaoqt9frUv6l6a0BCC7oFj+6IQIEYvFwvHjx+X/YAVprTl+/DgWS/B7Idaa\n3jtOY7s3c73OKbQSE1nriihEjdeyZUvS09MJ1nQrdZHFYqFly5ZB32+tzoi/7sngzE6NQx2GEHWO\n2WymTZs2oQ5DeFHrqnfcbTt4KtQhCCFEjVIrk/7yO4YD0DxeRuUKIYS7Wpn0G0SbAbj9veDO7CeE\nEOGuVib9OIs51CEIIUSNVCuTvsVsDHUIQghRI9XKpA+Q1LBeqEMQQogap9Ym/bO7NgXg6KmCEEci\nhBA1R61N+l9usk/DMPfLbSGORAghao5am/SjHCNxcwuLQxyJEELUHLU26Uc7kn52gSR9IYRwqrVJ\n/95zOgEwuF2jEEcihBA1R61N+l2b258+bzHX2iIKIcRpq7UZMdJk76ufW2gNcSRCCFFz1NqkH2Gy\nF23pJnmYihBCONXapG80KNo0ipb59IUQwk2tTfoAbRtFkyNdNoUQwqVWJ/0Yi4nsfEn6QgjhVLuT\nfqRJ+ukLIYSb2p30LSZOyZW+EEK41O6kH2GioNhGYbEt1KEIIUSNULuTvsXecydHqniEEAIoR9JX\nSr2mlDqilNrsY/1IpVSWUmq949+s4IdZMTEy/44QQngoTyf2N4DngQV+tvlBa31uUCIKoljHlb7U\n6wshhF3AK32t9SogoxpiCbqYSPuzcqWvvhBC2AWrTn+QUmqDUupLpVQ3XxsppaYppdYqpdYePXo0\nSIf2zVmnL331hRDCLhhJ/zegtda6F/Af4BNfG2qt52utU7TWKYmJiUE4tH8NoyIAOHwyv8qPJYQQ\n4aDSSV9rfVJrne14vRQwK6VqxCT2TepHArD5QFaIIxFCiJqh0klfKdVUKaUcr/s79nm8svsNhgij\nvXhv/7w3xJEIIUTNELD3jlLqXWAk0EgplQ48BJgBtNbzgIuBvymlioE8YJLWWldZxKfBcS4SQgjh\nEDDpa60vD7D+eexdOmuklNYNsNaMc5AQQoRcrR6RC9AgOoL8IpmGQQghoA4k/ViLiUNZeaEOQwgh\naoRan/RjIk2cyC2iyCpX+0IIUeuT/oKf0gD4fMOBEEcihBChV+uTvpNMryyEEHUo6dukA48QQtT+\npH/nmI4AtGxQL8SRCCFE6NX6pD+4fQIAMk5LCCHqQNI3GuzZ/vlvd4Q4EiGECL1an/QNjkv8NbvD\n8pEAQggRVLU+6UutjhBClKj1SV867QghRIlan/SFEEKUqPVJX6p3hBCiRK1P+g0cj0wUQghRB5J+\nq4SoUIcghBA1Rq1P+kIIIUrUiaQ/unNjAPKLrCGORAghQqtOJP3B7RsBUCAzbQoh6rg6kfSdPXje\n/WVvSOMQQohQqxNJf8XWwwDM/XJbiCMRQojQqhNJf2KflqEOQQghaoQ6kfRjLKZQhyCEEDVCnUj6\n7k7lF4U6BCGECJk6kfS126xrz67YHrpAhBAixOpE0nd3Ileu9IUQdVedS/of/pbOnmM5oQ5DCCFC\nos4lfYCRT60MdQhCCBESAZO+Uuo1pdQRpdTmANv1U0oVK6UuDl54wdGzZf1QhyCEEDVCea703wDG\n+ttAKWUEHge+CkJMQdc8vh7f3D3CY9n+zLwQRSOEEKETMOlrrVcBgZ4qfivwIXAkGEFVhWKr54MT\nh8z9NkSRCCFE6FS6Tl8p1QK4EHip8uFUnWJb2cnWth06GYJIhBAidILRkPsMcK/WOuAUlkqpaUqp\ntUqptUePHg3CocvPYjaWWbb9cHa1xiCEEKEWjKSfArynlNoDXAy8qJS6wNuGWuv5WusUrXVKYmJi\nEA5dfu0SY7hyYCuPZRk5hdUagxBChFqlk77Wuo3WOllrnQwsBv6utf6k0pFVgdQezT3eP/TZlhBF\nIoQQoRFwJjKl1LvASKCRUiodeAgwA2it51VpdEFmMdfJYQlCCOESMOlrrS8v78601lMqFU0VizSV\nrdcXQoi6pE5d+kaYVJll3WYtY87nW8gtLA5BREIIUb3qVNJvVr9emWU5hVZe/98e/vXVXyGISAgh\nqledSvrRkb5rs15dvZsvNx2sxmiEEKL61amkH8jrP+5h+gcbSD+RG+pQhBCiSshzBN38sjuDX3Zn\ncOhkPm9NHRDqcIQQIujq3JX+pH5JAbf5YfsxudoXQtRKdS7pz53Ykw9uGhRwO3msohCiNqpzSR/K\nzrgphBB1RZ1M+labJH0hRN1UJ5N+kZdplkv7YF06e4/nyqAtIUStUieTftM4CwB3ndXR73bDn/yO\n6974tTpCEkKIalEnk36XZnF8dedwbjmzfcBtf96VQZE18J2BEEKEgzqZ9AE6NonFYFB8cvOQgNu+\n8sPuaohICCGqXp1N+k69k+IDbnP4ZH41RCKEEFWvzid9gOEd/T/F640f9/DTzuPVFI0QQlQdSfqU\n72r/g3X7qiESIYSoWpL0y6nIqtFa+vcLIcKbJH3AqMo+XKW0zzcc4L1f5WpfCBHeJOkDU4e1Kdd2\nyzYfquJIhBCiaknSB2IiTeyZm8r70wZiNPi+6v/+r6NsOZBVjZEJIURwSdJ3M6BtAjsfHe93m1ve\n+b2aohFCiOCTpH+adh/LYeGmhaEOQwghKkSSfgXc9OmDPLzi/VCHIYQQp02SvhdzJnQrs0xTxCnj\nEjQ2ErL/w6srYvht74kQRCeEEBUnSd+LawYns2duKv2TG7qW5Rp/JiPiJTJNb7iWHTlZEILohBCi\n4iTp+7HopkHQ9GaKOU6UdSixxamcNH/EKeNSAPx09BFCiBpJkn4Aj4x+hFPRj6NQNCiaRj1rPzLM\n88gz/IqhHIO6hBCiJpGkH8DkHpN58YL7AVAYaVR4DxG6DUcjHmdn1qYQRyeEEKdHkn45TO4x2fXa\nQD0SC2Zh0LHMXHUV6SfTQxiZEEKcnoBJXyn1mlLqiFJqs4/15yulNiql1iul1iqlhgY/zJrFRAKN\nCx/iRF4Wqe+kcrLgZKhDEkKIcinPlf4bwFg/678BemmtewPXAa8EIa4aL0In06jwfv44+geXfnAp\nRdaiUIckhBABBUz6WutVQIaf9dm6ZM7haKBWzj+84q4RjOnSxGNZPdsZvDDuRZbvXM4tS2+RqZeF\nEDVeUOr0lVIXKqW2AUuwX+3XOu0bx/DKNSmsuGuEx/K96QO5f+j9zP9tPk/++GSIohNCiPIJStLX\nWn+ste4MXAA87Gs7pdQ0R73/2qNHjwbj0NWufeMYj/dv/7yXh0c9zKTuk7h3xb0s2rIoRJEJIURg\nQe2946gKaquUauRj/XytdYrWOiUx0f9zacOJQRl4/fzXGdpqKFd/fDU/7vsx1CEJIYRXlU76Sqn2\nStlHKSml+gCRQJ17irjFZOGTyz6hVf1WTHh3AjsydoQ6JCGEKKM8XTbfBX4COiml0pVSU5VSNyml\nbnJsMhHYrJRaD7wAXKZreYtm39YNPN5bbfbiJkQlsHSyfYqG8QvHczy3zp37hBA1nApVfk5JSdFr\n164NybErq6DYSqcHlrnet02M5tu7R3IgM49f92Rgrrediz4YS78W/fj6qq+xmCwhjFYIUZsopdZp\nrVMq+nlTMIOpKyJNRo/3u47mkFtYzOC537qWLbhqAZctvoxrP72WhRctxKBk8LMQIvQkEwXJuc+t\n9njfo+FY5o6ey3ub3+PBbx8MUVRCCOFJrvSDZNexHI/32QXF3DPkHnad2MWjqx+lbYO2TO0zNUTR\nCSGEnST9KqIApRQvpL5AWlYaN35xI63qt+KsdmeFOjQhRB0m1TsVNGVwcrm2MxlMLLpkEV0Tu3Lx\nBxez6bBMxyyECB1J+hU0e0I33r1hoM/1+UU2bI6unHGRcSy5YgkxETGkvpPKgVMHqitMIYTwIEm/\nEga1S/C57spX1/DApyWzUSfVT+KLy78gIy+D8949j+zC7OoIUQghPEjSr0LvrNnrer1hXyYXPnuA\nf415g/WH1nP5h5djtVlDGJ0Qoi6SpF9Npr75KwC23DN4ftzzfPHXF9yx7A6ZjlkIUa0k6VdSxyYx\nftcnz1jCrE83cyy7EAClYEqvafRteDXP//o8z655tjrCFEIIQJJ+pS2/Y3jAbRb8lOZ6nVtQzKK1\n+zi6/2I61z+Lu5bfRfKchyi22qoyTCGEACTpV5pjgtFye+7bHVhtGoWBi5Ifw6I7sVc/zqq0n6so\nQiGEKCFJPwhaJ0Sd1vaL1u4DwGywkGR7CKOOZ9KHF7L7xO6qCE8IIVwk6QfBV3cGruJxt+3QKdfr\ngsJYGhfOpshWROo7qZzIOxHs8IQQwkWSfhCUnnWzvJZsOgiAWSfR0TSHHRk7mLhoIoXWwmCGJ4QQ\nLpL0g+T20R2YfV5XbhvVvtyfySkodr0+fLQtL6W+wnd7vuOGz2+oVFfOTelZfPRbeoU/D/DL7gz+\nt+NYpfYhhKh5JOkHyZ1ndWTKkDbEWszl/kxuoefgrIcXJXBbv5ks2LCAOd/PqXAs5z2/mrsWbfC7\nzcGsPIr89Bi69L8/MfmVNRWOQQhRM0nSD7IWDepV6vOr1w0nungMc76fw4INCwJuv2zzIfo/soLC\nYhv5RVaycosCfuZkfhGDHvuWWZ9ucS2b8vovPLZ0a6ViF0LUfDK1cpCN696U1glRpB3PrdDnj+cU\nkcDNKNMxrv/selbv3sFXf65jb/6XtKrfikdGP8LkHpM5mJXHN1uP8MJ3OzhyqoBj2QXc9PY6NqZn\nBTyGs1rp222HgR4ArPzzKCv/PMp947tUKG4hRHiQpB9kSil6J8VXOOkDKMw0yJvBYct0Xl3/b5oW\nPIW2LCU98yTT30vn202LOZbRlA2lEnzphJ92PIfWCdFl9m9wjC2QGSCEqHukeqcKTOzTstL7MBBD\nYsEcwMzhiNkYbPEkFjyAydqWbzbWY/OBkwH3cTzHey8g53iyI6cKOOfpVRw5mV/peKvThn2ZfLp+\nf6jDECIsSdKvAsM7JrJnbmql92PSjWlc8BBa5WIzZHLIcjf76l1KuuUaDphmkmH+L6eMS8k3bOTj\nDX+g8bx0X7b5kKMKx5OiZBTxn4dP8cG6yvX0cff5hgMsXJPmsSy3sJjM3Ip3Qz2RU8gDn2yioNje\n8H3+C//j9vfWVypOIeoqqd6p4SJ1B1rmv02xOkSR2kuRYR9Fah9FhnSyDSvQpjwAbll5PwZLLGZb\nEmbdErNO4tnVSbz4QxI7/nk1J/OK2X4kmyHtG1F65ojnvtnu8/hr92SQktzQ9X7HkWwSYyKpH1XS\nS0lrzfIthzi7a1Nuffd3ACYPaO1af/bTq0g/kVfhE+ETy7fx7i/76Nkinkv7JVVoH1abpt39S5l+\nTiduPrP83WqFqG0k6YcBhRGzboFZtwDbINdyjcaqjtlPAmqf44SQTq5xDTb1lWu7hCdvpbigGWZb\nEvedNYYWse0pUicw6WYoTBQUl3TdnPf9TuZ+uc31/pyXHyI28XNXA/KYf39Pq4ZRrLrnTNc2i9el\nM33xRuZM6OY1/vQTeR7v1+7J4JkV23nj2n6YjIFvNq2OJ5CVvpMpL5tNczDLHsOz32wP66T/6fr9\nPLJkKz/dNxqj4fTmfRICJOlXqV5J8WzYl1ll+1coTDoRk06kHn3Ardu/lSzXSeC8HjbeX/8z+YYt\nPPDdSvsGFkAbMelmmHWS4w4hiTnLdmCmJQYsrn2lZaVx42e3UVQMEM/eDM9G6sOONoHN+0sakg9k\n5lFYbCO5UdmG5Ivn/QTA/sw8rw3NvpRueNZa8/6v+7iwTwu/o6Jf+n4nTy7/s9zHqUqHsvJpWt8S\neEMfHvh4M6cKiskuKKZ+vfKPCRHCSZJ+Ffr05iEkz1gSkmMbqY/RVh8L3Vn5CzRhPAA28ihS+92q\niew/80xrQJVc8RttjTHrJMDevdOcdy53L/4BgzEeg47l481xPPRJGh/deJarC6h728Dgud8C8IPb\nHcGOI9lEmkqu7N3bFpze+H0hs1fOZO/Jva4uqsrRrbS0LzcfYsZHm0jLyOXesZ19/i6+/+toyZsq\n7LGktWZ/Zh4tG3ifgO9/O44x+ZU1zLuyD2O7Nwu4vyKrjYOZ+bRym9DPWTUnD98RFSVJv44xUI9I\n3Z5Iq2cVh6aIInXQ40RQbEjHqjJBKzLNngPFLvrQ/rPDi2DACJZoDDoWAzEYdSwGHYOBWHr8eyEG\nYywGYhn8zK8YicGgYjHoWKy6mLxCK2kZOYx95gfaNCli9+F4MkxnoM1ppGWlMe3zaXRX7wJGsguK\n2XqwpNdSVp59IFpGdiE2m0Yp71Ndu59ofEk7nsOm/Vmc27P5af5G7d5Zs5fj2QX86+u/WHLbULo1\nr19mmz8cPa5uevs3gIBtHHM+38LbP+9l3QNjSIiJBErKV9Gc/9PO45zML+Lpr//i478PoV5ExeaN\nEuFLkn4VWz/rLC568Ud2HcsJdSh+KcxE6FZE6FbgZXYGTSFWsrGpU/Z/ZGNT2YzsXI8/Dh9gX9ZR\nbJzCprKxqiyKVLpjW9/lbvsCKF0Pg47FGBnL4RMxGMyxuF+OFxTUY0fxcpQxkvu//BFFJMpgwaAt\nHMyOoZjjvLs2h/fW7mFM52a8OqVfmeNEuLUbaDTFVhu7juXQsUksYG8zGPHkSgAmfNzSYxCc09Nf\n/0XLBvW4JCWJFX8c5voFa1EKdj+Wyrq0E9z/8SbXtlv2n6Rb8/r8uPMYJoOB/m3sDeHe6uAXblrI\nzG9msjdrb5nj/rDdPvfRyfxit6TvLEfFXP5yyXMbth46SZ9WDSq4J7tlmw9RbLNV+GQpql/ApK+U\neg04Fziite7uZf1k4F5AAaeAv2mt/U/8UofER0VgLkdjZU2niMBEQ9ANPTLOWsdMDgk+PqexYiPX\ncaKwnxScJ41rhycyf/VGx/JT2MihyJCGVZVMPW01HCcj4kWv+777B8Bt1ovX9ph5Y7aFxjFxxNeL\nJSYiBosxio378ik2R6CwYFIWRr/6MRv25tGvdTNGdEjinV92k2uIRmFBa01aVhrXf3Y9R3KOMKnb\nJCwmC09/sxlFBIPaJXD9grX2sjl+D1l5nt1RbY4VV7xsn7to8U2DuOnt3xjUzvO3tHDTQqZ9Po3c\nolwMOtp1ZwNwRfcrvF7Nlwys0yxel85rq3ez9PZhPn77Jdret4S7z+7kseyi9y5ib973Xk9y5XXT\n2+sAfCb9YqsNDWH7f+CPAydRCro0iwt1KEGjAtUNKqWGA9nAAh9JfzCwVWt9Qik1DpittR4Q6MAp\nKSl67dq1FQw7vLy2ejf//OIPPrhpEJc4GjFL2zM3NWT1/6HyvxmjGOKo+y8trd65RBePpWHRVDT5\n2FQ+mny0yseG/fXElEQWrduBVnmOZQVolU+zeEVMvWL2Z2WSmXfK7TP29agCbFRs3ICBCLQ2ozCj\ndCQdGjegqNjEvowilI5EYaZH80a0T2zIZ+uPoYhAabP9p/OfNtvvWOov5HjecSKtvalffAnHzc9h\nNRymWUwzzmz4Et//meX4jJkJPVvz/BUp9H34a47nFPLLzNH0f+QbwHc10bq0DGItZjo2ifX6t3Uo\nYjoFRvt8S1HmKOafN5/JPSaTkVOI0aDK1VDs3K+vGAY99g3Hswv565FxrmX+7m6qUkWOG6h8oaCU\nWqe1Tqno5wNe6WutVymlkv2s/9Ht7c9A5Yej1jLXDW3DtUOSPeqbkxrWY19Gnp9P1X7/8TM+ACCu\neAIG6gFDetIJAAAaxUlEQVT1MDqvTdyuUQY3786Xv2wu87mCo1CAvYNS01LrzEZFz5bxrE07Zj+Z\nOE4kzhNLpukdCoz2G9W4oomYdFP6JEezNu2I/aRBEVoVorH/69mkEftOZLH/+BE0hdhUHodzczhx\nYCf5hkyPbd0bygFwfP0FxvUcMZYMNjuYfZB3si/wuIt5YTvMf9hMsdWEskTQ8fl65EQaUETQ4dmZ\nZGRr+rZqgsVkwWKyEGm08MnvR1BEcOOwTmSa0l0nEKXtP62qpH0ktyiXu5bdRfOY5lz+8i8YlZlP\nbx6OyWDCbDBjMpjsr41urw1mbOQCRtalHeNAZgHn9WrhUcSDWZ6jvRduWsjdi9ZzUjVDG9M87m6C\nkfhfXrWLzzce4LNbhpY5rvOuCgj6cf0J1UnOl2DX6U8FvgzyPmuF0g2ML03uy7n/WR2iaGqG937d\n53NdpK0TEYYIj26opW0/fMr3Sh+0tle/KIwoojEQXXIi0WDSDSlwvG1gGAiFXdi+E8o2y9q9f3Eq\nX/9xmBsWlNy1PjG8JxP7tKTd/Us9j43VdQLQqhDdaCYHsg9Qv+gqLLauZJoWUmDcTKPCex3bFZVs\nSxF/G5nEC99vRVNEHFBky0GrQtKP29dnF2ZzLPcYecV5HM/JJs+Yg6aQ59Z8TaE58FQbR3KPMGrB\nKLA3H9Dv5XL8Qh0nppQ37D+NnxpdJwebzUC+BRQmWj0djclgYt/JfShDc9wnA8gtyuW6T67j6Z+e\nxqAMKKVQKL8/DcrgdZ29HUQxbmFjFIrcQitGZeCn/d+TV+x5kZVblMvfvvgbvx/8nYxszeEsKyM7\ntiDSFOk4cUaSY/wDpc0s22Ek0uhYbor0eO3c1kAEv+zOpF1irKvH1dsbF3LHh0s4obLQSlfrycaX\noCV9pdSZ2JP+UD/bTAOmAbRq1SpYhw5LMrDGv6YF/yLWYuKUtdjnNgt+SvO5zpdim+b3vb7HTthU\nPkpHkGC7nDbxHdh9xP/+NqVneSR8gHsWb+T5b3eU2dZ+orHfuaDh/856ghu/mEZkQXssth5EW0dg\ntOwiOs97Hf0jo8bz7grHiSQLEkut/3FqSRXESyt38viykkF29oFtxW4nHfsJ5YDlJtc2TaKb0DXy\nQbYczASsvHLNGRTbilm94zBbD53glz1HASupPZvQv019im3FzPlio+NkZgWs3HxmGzbtz+DPw5kc\nPJlDlGP5mLbNKbIV8fbGt4nSSWisFLHHdexCaxHxkYmYTfb2Co32+9Ombdi0rcw6m8oFNMdzjZzK\nL2LXsWxAU2jwfld9qvAUL619idzCfFA2lu4ttUGE/ce4hV4/XpZWKCKIs9TDYrJwLPcEikQM2oLN\n8V8+tyiXmd/MDO+kr5TqCbwCjNNaH/e1ndZ6PjAf7HX6wTh2uDK4Xfk/MbEnAOd0a8LyLWXnyqmr\nTuX7TvhVpXHhTIheCvnjAyZ8sD+wxpvSA9i8eeCdeC7ssoDV2+xdT2Ot43juvEHM9JFgrAHa37TW\nKKXIKSgm7bhnryn7mAh7WwRudzcWa18ibR1Q5sMknLqTXccMrlqlfk1H0ygmgjvf/BJIIsax/Pvf\n4c3L7CeYpz/xbCtIbT2AhV/bG7Abui3fsz2aC85owQ9xP8Dh+wB7u41Ta/1Pdmw9g12PjsdwmhdE\n69JOMPGlH/ni1qGuu+cjO+3rXKMhmt5MWpbnRYJBx5EU34A9d+whecYSNFa2/HMUBcUFFFgLyC/O\nZ9gTX6Ep5KOb+5NfnE9BsX15XlE+t7y3hqSGZq4d2oJCayFv/bydtIwsNEXYcoooNhWj9Q9E2trY\nT7I2jdVg//+9N6v02aX6VDrpK6VaAR8BV2mt/6p8SHWD8++6SVykaz6Z/1zeh44PSO1YyOWMr7ZD\nrd5qAbfRz5d2vZyZPmpIbQGTPtyzeMNpTaDXpNDxhLZiKN25duBj3/DR3wd7/dzHv6eTXVC27u0K\nH09b23Ush39//RcPX/EID77juS7KHIXhVG9s2E9sBrdBe8ezCzAZDR6NygXFVoxKuabwmPiSvVnR\n1wkY4IYec3l0zVRXnb7F2psmhf/HsAYl1V4KI4t/Pcq6vZl8vuEAr0/p5xigCANbDnRtlzxjCWO6\ntCXGGsOJo7B8TRxLbhvGhq1ryDri9ojRIoiJi4OTVwGQaXqHLIO98K3qh66mozxdNt8FRgKNlFLp\nwEOAGUBrPQ+Yhb3H3ouOeuviyrQs13bxUWYyc4vKTHoGEOFnEFGvlvXLzJ/vblK/JL915CI8dJjp\n+6T/6e8H/H52x9HsoM6YCrDtoPd2kzvfr1iv7Mu7Xc6DjpOaQrkaNme/Z8Bm1VhtGrMRHvxkM0dO\n5bN8y2GMBsXOR0tOxJ0eWEb3FnFc0jeJx74sedqbv3Pihz8mMP+C+a4G1SYR/aEQftjqOSXG7M//\ncL2+9o1fPdZl5BTy+v92A7Bia8lt4BbHoDvnuAoPjoRvL6/9/3eUOYpHRj/iO9gqVp7eO5cHWH89\ncH3QIqrlFt04iC82HCAm0n7lUt6Rla9N6Uff/1vhdV3rhCgm9GouSb+Wu+fDjX7XL910MOjHPJZd\nEHij05Dh9oyHZ4b+ybPfbIfcbhjUH4Dmsw0HuDQlibd+LqmKsdo0Npvm++1HGda+EQCb959k8/4t\npXfv1+Qek1316C98t+O052Pq8/DXPteVZ1oMg46mdf3Wta73jgigY5NY7jq7k2uSsvI2bDhHZPok\n7cJ13jMr/HeBrYh/fx3cGtv+j35TZt8zPy7pdnvP4o28tnp3mc+9vSaNWZ9uoVdLX/2o/Cv938Pg\n7Va7Et7+OXCnglsH3MrsCfOCetyKkKRfCwT6803t0YyT+UXebz+FqGG2HSpbpeScnttfFac/x3MK\nSZ6xhM5NY3kgtavX6lV/Ag2cfPDTwHcdxTYv85uEQHiOja4FnBNdOedlcfrn+WXnpO+f3LDMstJd\nPlvGe5/ZEeCqQa2Jj7L3PevkmG9GiHAyf9WuoOxn26FTXPnqGr7YWNI+kuHjsaLBZq0ZOV+u9EMl\nzmLmqzuH06qhZ7K+elAyC3/ey5+OgUcXndGChy+wz36xZc45bNqfRYv4eiQ1jHJdfSilPKbfLc2g\nSiYxjrFU/Ctv2yi6xk8cJ0R5bN5fMhrZX119MFnlSl90bBKLxVx2att/X9bL9fqJi3sSHWlP1NGR\nJga2TSDJcaK4Y0wHj88lREeU2dcNw9qQ0rqB63bWW7e/S1Na8vktZcfULbpxkMf7YR0a+S3P3Wd1\n9LveXWu3k1Tj2ADtFQ4/zhjF2G6lJ1YQIjwU22rG0CRJ+jWQ+1zs/kbuXnSGfZojZx9mb4N3ZqZ2\nxWBQnN/bPgtir5bxZba5oHeLMncc4Fn19NN9o4i1+J+A64oB5et7vGduKoluDdPjezQr1whlk0HR\nvnFMwO3CgdkoLe91jVWSvigPbw8FcWqVEMXs87oy/6q+QMkf1RMTezKsQyOendTbte2ozk3YMzfV\na3JvUt+CcvtLuLhvS3oleZ4cmtWvR6OYsncS7kwG739OG2efXWZZ6UcGnt/Lc2rexTcN4qIzPCfv\nQnHaDXBVpa2Xx0CeDl+/K1F7yZW+CIopQ9rQOM6eQJ1Jf3zPZrw1dQDn925RZvvUns1oGmdh6tA2\nADw7qTftEmM8urA9dUkvPr15CABvXtefl6+2j7W7cmBrHrmwzOzaLr7yWJyXO4THLvJ8BGJ+sefo\nzpTkhvz7st4ey4zK2wMWKyY2snLNWdc5fn/lcXn/sndAeUV+ZpITtZLVKklf+LHoxkFMP6dT4A3d\nOJO+yU9VSZM4Cz/fP5oHUruwavqZrhODr4+M6JjIWV2b2PdrNDB5QGuv27VLjKael/YJp4FtPXsg\nla4qyissSYJPXdILb6IjTeW61G+dEEVqj2bER5U92bSIt88sU9mrrgZRZe96vPWyAkL6APMHUruE\n7NjCk1zpC7/6t2nIzWe2D7yhG2cjbXkGnpTu8ROMwSomo8FnQ+t70wZ5Xe40oG3JU6Uu7lvySIYp\ng5NdryNNhnJd6X95+zBemNyHr+8cUWZd4zjPxw5W1PgeTXni4p4ey96bNtDrttcNTXa9HtkpkS/L\n8aQrf96a2r/c2zaPrxd4I1EtpPeOCLobhrUF/F/p+xKsuvJ5jvYFbx69sIfX6qE4i4lpjthLe/Dc\nrq7XSimuG9qGsd2acuXAkiqTDQ+dzYVu9f/OE1hiqV5BfVrF89+r+vL4xB40ifNsUzgd3ZrHoZTi\n0pQkj+UGg+Ln+0aX2T7SZL8Dqmc28sa1/Sv96L2h7f33onKaOrSNDNSuQeRKXwTdPWM7s2du6mlP\nTQvBHZb+zd0j+PBv9iv7SLdJ5K4Y0Mqjemjj7LO5ZlBr/n5mewwGxfvTBvLVncM99lW6V0/9embm\nXdWXhOhIj2VPu9X/u5eljaPBtXl9CwumDqBxrIXL+rUK2JNi9nldPd73aGHvUTWmS2OW3FZypd6x\niWdvotIN1BDcxufOTWP9Nu4DbP3nWHY9Op4Hz+16Ws9tuH10h8AbVcCsc7vyy8zR/PHPc1zLRnVu\nXCXHqskCzZJaXSTpCyC4Sb9dYgx9Wzfk++kj+XHGKJ/bxVnMzDm/u2uswoC2CXT0MmL4478P9uiJ\nFIh7onPeWbRKiCLGrfH27rNLxhS0bFC2CmTyQM+2i89vHcqzk3rzr0s945h+TmefcTjniXFG4+tX\nfO2Q5DLL4nwMont1Sj+vy917E5mNynXiL/1Acm/tLhGObSb1TyqzrjxuG+W/GvK6oW1oHGshKqKk\nTGO7NWXlP0ZW6HgAk8vZPbgmKZKGXFGTnO7NQXn6y7dOiA48UVw5nNGqQZmeSN4SaLKjjaI8ZTm/\ndwt+uOdMr+sSoiPKJEvnZ0o3yroaub0c9J0bBrL63jO9Xpk7p8MY3C6Bh84rO/VGhKlscr5yYCtX\nQ7STM3HOcrszcT/p9XCceF6bksK2h8cSHVl2v9ox7V98vZLG6benDmCTl6623tx1dievje8dGsfw\n9tQBXj9j05pkL91el90xjIl9Stp0fJ0oT+cipbJtKMFSU/rpyzQMAigZD1Ce9oD1s84i0mSky6xl\nrmU14c950Y2D2LQ/K2D1h5MzsTeJs7gm9Fr3wBgi/fRC8mbZHcNo6KU3T3SkiehIEzkFZZ8A5pwO\n4y4fo5ib1o/kWHYBURFGch09m7x1WE1uFM2euakey9zL3ygmstR6z30YVMn03u6/tqF+Rl+vuX80\nA9xmywQ4t2cz/vFByRz77RKj+eTmIa7R5KX5+nvp3DTOY7S28rFtVET5v6MEt/ElVw9qzS+7M7xO\n6lZVeiXFs2FfptTpi5rn3rGd+fxWn484domPinBNGBcq3vrJN46zMLpLE49lzq6V3qqNmta38O9L\ne/Hfq/qy9oExbJp9NgkxkR7VQOXRuWmca6wEQEykKeDUEs4pLZwNyqXvOqYMbsOeualc5VbNNK5H\n5aegKH1OH9W5sSupGg2KORO6seS2wH8DZfdbsuPrhrTho797T/jOMQvO+m1v3WrLU/V986j2zDq3\nK2Mc33dXP43jjWMtXOAYkT4ztQuzzu3qc9vT9YljPIs/lzka/Cs7oC9YJOkLl7+NbFfpniXVJc5i\nZkTHRHoGmF+9S7M43ps2kJk++qtf1KcljWIiaRQTWWbsgLOaZPkdw7191Kf1s87y25YBcNuoDvw4\nY5RrHqUkt5HS2x4eW9Jt1ZFL7x3bmcHtytdrx58Hzu3qcVJ7/oo+ritrg1JcMzjZYxoQb7zNF+V+\nMpl1XlefYxPaJdoTX1PHye7n+0bTvFTj94Z03w+ud4qzmLluaBtXR4GbRrajX3IDx7qyJ5tnJp3B\nnrmpRJqMDG7fiB2PjPO7f2dHhEC8zXfl7qXJfbhiQCssZoOru3CoSdIXYevN6/rzmZeJ4kob2DbB\n1W3ydKz8x0jeuX4AnZqe3nTUJqPB9fxW8F49YTAon33ovSXVYJnQqzmb55T0orGYjbw3bSCvXJ3i\ntaePM0k7DevQiPr1zCy7w7OevLxVatcNacM7Nwxw3ZFZzEaWlqpzd/6+I02GgPstcsxXHGFUvDV1\nALeNal+uq2+T0cBoLz2I/vy/sayfdRZ9W5cMtPPW0O9UL8JYZuJDdx0dZTEZDDVmRK7U6YugqI39\nwZMaRnlcgVdWoN/R1KFtyv34zGBqHGthTFfv4xbcw5kzoRuXpNjvQDo39bwjLG9HAINBlbljiY+K\noG2jaFdj/e2jO9ClWRzn9WzmemZwu8Rodh4tO623M+mbjQYsZiN3nW0fxX7FgFa8s2av31iev6IP\nAx5dwcl8e5vL2G5NiTQZXRcIXZvF0aphFCfzi1xtPk7ndGvCbaM70Cgmssw8Ve6c3+eQ9gleG65D\nQa70hahizhk1R3RK9Lvdg+d29eiFc7oqO5+QN+4noXaJMR7dLt05r8gr+pCeb/8xktsdV8wWs5EJ\nvZp7XOUvvmmw1885G0dL97aafnbgKUzqRRg5122ivwalqmqW3j6MeVf19dq/PsJkdFWDjeyYyIxx\n3rvuOqvS/ntVClcO9D6FSXWTK30hqlikyciq6WdWeZ3uyukjycorCuo+3RNeoBqcRTcOCvrU185j\nGpRiyW1DSX1utcf6h87rxiNL/ijzBLrSCdyXB1K78MeBk6zfl+nz4ebeOt24DzpUSnHTiHY8vmyb\n6yS55v7RHMrK9zpYL9Qk6QtRDfw92SxYEmIigzIuwp37lX3ppP/alBSSGpSUq3TiDSaDwf6cif/N\nGEW+2wyl7RvH8Pq15Z+LqLSoCBOX909i/b5MnyNm3U8Gt5zZnpP5RdwxpmxX26/vHM5LK3fx6EXd\niTQZKzXVR1WSpC9EHbTirhFsORD4IeMvX92XoY9/B0BspGePnFGdm3j7SJVwNjKXHpzmzzOX9Wbz\n/sBldFYj+epGf16v5vy65wQANwxv67NnUvvGsfzrUu8zxNYkkvSFqMEax9qvFgN1DTxd7RvHlKsq\npmWDKLbMOYfv/jziGt1bnZRjeFZFpgm54IwWXFD6QTxeOPv4D2mf4HX9VQNbc2lKEnmF1pBOkx0s\nkvRFhXVuGsv+E3mc8jLiVATHlMHJJMZGcm6PZh7Lf3/wrApNrFcR0ZEmzu3ZPPCGVagqn5jWvUV9\n1s86i3gvo6rtx1ZYzMYq7UpbnaT3jqiwZXcM55Nb7H2ia8t/iJrGaFBM6NW8TIJvEB1RK646A7ln\nrL0XjrmKHy/pK+HXRnKlLyqlbaNo7hzTkYl9A99GC3G6rh/Wlut9PGtBVIwkfVEpSilX/2ohRM0n\n1TtCCFGHBEz6SqnXlFJHlFKbfazvrJT6SSlVoJT6R/BDFEIIESzludJ/AxjrZ30GcBvwVDACEkII\nUXUCJn2t9Srsid3X+iNa61+B4I7/FkIIEXRSpy+EEHVItSZ9pdQ0pdRapdTao0ePVuehhRBCUM1J\nX2s9X2udorVOSUz0P82sEEKI4JPqHSGEqEOUrzmkXRso9S4wEmgEHAYeAswAWut5SqmmwFogDrAB\n2UBXrfXJAPs9CqRVMO5GwLEKframkjLVfLWtPCBlChfuZWqtta5wVUnApF8TKaXWaq1TQh1HMEmZ\nar7aVh6QMoWLYJZJqneEEKIOkaQvhBB1SLgm/fmhDqAKSJlqvtpWHpAyhYuglSks6/SFEEJUTLhe\n6QshhKiAsEv6SqmxSqk/lVI7lFIzQh1PeSml9iilNiml1iul1jqWNVRKfa2U2u742cCxXCmlnnOU\ncaNSqk9oo7fzNuNqRcqglLrGsf12pdQ1oSiLWyzeyjRbKbXf8V2tV0qNd1t3n6NMfyqlznFbXiP+\nLpVSSUqp75RSfyiltiilbncsD9vvyU+Zwvl7siilflFKbXCUaY5jeRul1BpHfO8rpSIcyyMd73c4\n1ie77ctrWX3SWofNP8AI7ATaAhHABuxjAkIeWzli3wM0KrXsCWCG4/UM4HHH6/HAl4ACBgJrQh2/\nI67hQB9gc0XLADQEdjl+NnC8blDDyjQb+IeXbbs6/uYigTaOv0VjTfq7BJoBfRyvY4G/HHGH7ffk\np0zh/D0pIMbx2gyscfz+FwGTHMvnAX9zvP47MM/xehLwvr+y+jt2uF3p9wd2aK13aa0LgfeA80Mc\nU2WcD7zpeP0mcIHb8gXa7mcgXinVzNsOqpP2PuPq6ZbhHOBrrXWG1voE8DX+p+6uUj7K5Mv5wHta\n6wKt9W5gB/a/yRrzd6m1Pqi1/s3x+hSwFWhBGH9PfsrkSzh8T1prne14a3b808AoYLFjeenvyfn9\nLQZGK6UUvsvqU7gl/RbAPrf36fj/8msSDXyllFqnlJrmWNZEa33Q8foQ0MTxOpzKebplCJey3eKo\n7njNWRVCmJXJUQVwBvaryFrxPZUqE4Tx96SUMiql1gNHsJ9UdwKZWutiL/G5YneszwISqECZwi3p\nh7OhWus+wDjgZqXUcPeV2n6vFtZdqWpDGRxeAtoBvYGDwL9CG87pU0rFAB8Cd+hSU6KE6/fkpUxh\n/T1pra1a695AS+xX552r47jhlvT3A0lu71s6ltV4Wuv9jp9HgI+xf8mHndU2jp9HHJuHUzlPtww1\nvmxa68OO/5A24GVKbpfDokxKKTP25LhQa/2RY3FYf0/eyhTu35OT1joT+A4YhL16zeRY5R6fK3bH\n+vrAcSpQpnBL+r8CHRwt3BHYGzQ+C3FMASmlopVSsc7XwNnAZuyxO3tFXAN86nj9GXC1o2fFQCDL\n7da8pjndMiwHzlZKNXDcjp/tWFZjlGo/uRD7dwX2Mk1y9KRoA3QAfqEG/V066nlfBbZqrf/ttips\nvydfZQrz7ylRKRXveF0POAt7W8V3wMWOzUp/T87v72LgW8cdm6+y+haKluvK/MPe2+Av7PVfM0Md\nTzljbou9hX0DsMUZN/Y6uW+A7cAKoKEuadl/wVHGTUBKqMvgiOtd7LfRRdjrDqdWpAzAddgbnHYA\n19bAMr3liHmj4z9VM7ftZzrK9Ccwrqb9XQJDsVfdbATWO/6ND+fvyU+Zwvl76gn87oh9MzDLsbwt\n9qS9A/gAiHQstzje73CsbxuorL7+yYhcIYSoQ8KtekcIIUQlSNIXQog6RJK+EELUIZL0hRCiDpGk\nL4QQdYgkfSGEqEMk6QshRB0iSV8IIeqQ/wdaDat1t+cZBQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10e3aa190>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#retrieve train and validation summary object and read the loss data into ndarray's. \n",
    "loss = np.array(train_summary.read_scalar(\"Loss\"))\n",
    "val_loss  = np.array(val_summary.read_scalar(\"Loss\"))\n",
    "\n",
    "#plot the train and validation curves\n",
    "# each event data is a tuple in form of (iteration_count, value, timestamp)\n",
    "plt.plot(loss[:,0],loss[:,1],label='train loss')\n",
    "plt.plot(val_loss[:,0],val_loss[:,1],label='val loss',color='green')\n",
    "plt.scatter(val_loss[:,0],val_loss[:,1],color='green')\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* plot the MAE curve during validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-03T09:51:03.691390Z",
     "start_time": "2017-08-03T09:51:03.436382Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt0XOV57/Hvo7tkXSyN5atkW8I2tsGAQbYEDgGSYkxa\n4LTnlJjcIJRCVgonpQldJFlJCFlNk7ZpTpqy0pBAgVBwgJTUtDQGAgRIfQfb4ItsYxskG1uyZV1s\nWdbtOX/MlhjLsjWWRh5p5vdZa5Znv/PO6H3XwG/veeadvc3dERGR5JAS7wGIiMjZo9AXEUkiCn0R\nkSSi0BcRSSIKfRGRJKLQFxFJIgp9EZEkotAXEUkiCn0RkSSSFk0nM1sC/AhIBX7u7t/r8/gPgauC\nzRxgvLuPjXg8H9gC/Nrd7zzd3xo3bpxPnz496gmIiAisX7/+oLsXD9RvwNA3s1TgAeBqoBZYa2bL\n3X1LTx93vzui/13A/D4v8x3gtWgGPn36dNatWxdNVxERCZjZe9H0i6a8sxDY6e673L0dWAbccJr+\nNwFPRgzkEmAC8EI0AxIRkeETTehPAWoitmuDtpOY2TSgDHg52E4BfgB8ZWjDFBGRWIj1F7lLgWfc\nvSvY/iLwvLvXnu5JZna7ma0zs3X19fUxHpKIiPSI5ovcvUBpxHZJ0NafpcBfRGxfClxuZl8EcoEM\nMzvi7vdGPsndHwQeBKioqNC5nkVkUDo6OqitraWtrS3eQxk2WVlZlJSUkJ6ePqjnRxP6a4GZZlZG\nOOyXAp/q28nMZgOFwMqeNnf/dMTjtwAVfQNfRCRWamtrycvLY/r06ZhZvIcTc+7OoUOHqK2tpays\nbFCvMWB5x907gTuBFcBW4Cl332xm95vZ9RFdlwLLXFdlEZE4aWtrIxQKJWTgA5gZoVBoSJ9kolqn\n7+7PA8/3aftmn+37BniNR4BHzmh0IiJnKFEDv8dQ55cwv8htbG3nRy/t4J29TfEeiojIiBXVkf5o\nkJJi/Oi32+ly5/wpBfEejojIiJQwR/r5WemcN7mA1bsOxXsoIiIjVsKEPkBlWRFv1TTS1tE1cGcR\nkWGwZ88eZs+ezS233MKsWbP49Kc/zUsvvcSiRYuYOXMma9asYc2aNVx66aXMnz+fyy67jOrqagC6\nurq45557WLBgARdccAE//elPYz6+hCnvAFSVh/j5G7vZUNNIVXko3sMRkTj69nOb2bKvOaavOXdy\nPt+67rwB++3cuZOnn36ahx9+mAULFvDEE0/wxhtvsHz5cr773e/y2GOP8frrr5OWlsZLL73E1772\nNX71q1/x0EMPUVBQwNq1azl+/DiLFi1i8eLFg16e2Z+ECv0FZUWYwapdhxT6IhI3ZWVlzJs3D4Dz\nzjuPj3/845gZ8+bNY8+ePTQ1NXHzzTezY8cOzIyOjg4AXnjhBTZt2sQzzzwDQFNTEzt27FDon0pB\ndjpzJ+WzeldDvIciInEWzRH5cMnMzOy9n5KS0rudkpJCZ2cn3/jGN7jqqqt49tln2bNnD1deeSUQ\n/vHVj3/8Y6655pphG1tC1fQhXOJ58/3DHO9UXV9ERqampiamTAmft/KRRx7pbb/mmmv4yU9+0nvk\nv337do4ePRrTv51woV9ZVsTxzm421mi9voiMTH/913/NV7/6VebPn09nZ2dv+2233cbcuXO5+OKL\nOf/887njjjtOeDwWbKSdNaGiosKHchGVxtZ25n/nRe7+g1n834/PjOHIRGSk27p1K3PmzIn3MIZd\nf/M0s/XuXjHQcxPuSH9sTgazJ+azSuv1RUROknChD1BVXqS6vohIPxIy9CvLQrR1dLOpVnV9kWQz\n0krWsTbU+SVo6BcB6JQMIkkmKyuLQ4cOJWzw95xPPysra9CvkVDr9HsUjslg9sQ8Vu1q4M6PxXs0\nInK2lJSUUFtbSyJfdrXnylmDlZChD+H1+r9cW0N7ZzcZaQn5gUZE+khPT4/pr1cTUcKmYWVZEcc6\nunh7b2O8hyIiMmIkbOgvDOr6q3RKBhGRXgkb+qHcTM6dkKf1+iIiERI29AEqy4tY/95hOrq64z0U\nEZERIaFDv6o8RGt7F2/rurkiIkCCh/7C3vX6quuLiECCh/643Exmjs9VXV9EJJDQoQ/huv66PQ10\nqq4vIpL4oV9VHuJoexfvxPhamSIio1HCh/6H6/VV4hERiSr0zWyJmVWb2U4zu7efx39oZhuC23Yz\nawzap5nZm0H7ZjP7QqwnMJDxeVmcUzxGJ18TESGKc++YWSrwAHA1UAusNbPl7r6lp4+73x3R/y5g\nfrD5AXCpux83s1zgneC5+2I5iYFUlodYvmEfnV3dpKUm/IcbEZFTiiYBFwI73X2Xu7cDy4AbTtP/\nJuBJAHdvd/fjQXtmlH8v5qrKQxw53smWD1TXF5HkFk0ITwFqIrZrg7aTmNk0oAx4OaKt1Mw2Ba/x\n/f6O8s3sdjNbZ2brhuOUqFWq64uIALE/8l4KPOPuvdcpdPcad78AmAHcbGYT+j7J3R909wp3rygu\nLo7xkGB8fhbl48boR1oikvSiCf29QGnEdknQ1p+lBKWdvoIj/HeAy89kgLFSWR5ize4GuroT84o6\nIiLRiCb01wIzzazMzDIIB/vyvp3MbDZQCKyMaCsxs+zgfiHwEaA6FgM/U1XlRbQc72Sr6voiksQG\nDH137wTuBFYAW4Gn3H2zmd1vZtdHdF0KLPMTL045B1htZhuB3wH/4O5vx2740assCwGq64tIcovq\nconu/jzwfJ+2b/bZvq+f570IXDCE8cXMxIIspodyWLWrgdsuL4/3cERE4iKpFq1XlYdYs/uQ6voi\nkrSSKvQry4tobutk237V9UUkOSVX6PfW9bV0U0SSU1KF/uSx2UwtytGXuSKStJIq9CG8dHPN7ga6\nVdcXkSSUdKFfWRai6VgH2/a3xHsoIiJnXfKFfnlw3dzdKvGISPJJutAvKcyhpDBbdX0RSUpJF/rQ\ns15fdX0RST5JG/qHWzvYXqe6vogkl6QM/crg/Po61bKIJJukDP3SohymjFVdX0SST1KGPoRX8aze\n3cCJJwUVEUlsSRv6VeUhGo62s6PuSLyHIiJy1iRv6Afn4VmtEo+IJJGkDf3SomwmF2Tp5GsiklSS\nNvTNjMryEKt3H1JdX0SSRtKGPoRPvnbwSDvv1quuLyLJIalDv+f8+itV4hGRJJHUoT8tlMPE/Cx9\nmSsiSSOpQz9c1y9i1S6t1xeR5JDUoQ/h9foHjxxn18Gj8R6KiMiwU+iX91w3VyUeEUl8SR/600M5\njM/L1MnXRCQpJH3omxlV5SFW7dJ6fRFJfFGFvpktMbNqM9tpZvf28/gPzWxDcNtuZo1B+0VmttLM\nNpvZJjP7ZKwnEAuV5UXUtRxnz6HWeA9FRGRYpQ3UwcxSgQeAq4FaYK2ZLXf3LT193P3uiP53AfOD\nzVbgc+6+w8wmA+vNbIW7N8ZyEkMVWdcvGzcmzqMRERk+0RzpLwR2uvsud28HlgE3nKb/TcCTAO6+\n3d13BPf3AXVA8dCGHHvl48YwLjdT6/VFJOFFE/pTgJqI7dqg7SRmNg0oA17u57GFQAbw7pkPc3iF\n6/pary8iiS/WX+QuBZ5x967IRjObBPwC+Ly7d/d9kpndbmbrzGxdfX19jIcUncryEPub23i/QXV9\nEUlc0YT+XqA0YrskaOvPUoLSTg8zywf+C/i6u6/q70nu/qC7V7h7RXFxfKo/l5aHr5ur9foiksii\nCf21wEwzKzOzDMLBvrxvJzObDRQCKyPaMoBngcfc/ZnYDHl4nFOcy7jcDJ1fX0QS2oCh7+6dwJ3A\nCmAr8JS7bzaz+83s+oiuS4FlfmJR/Ebgo8AtEUs6L4rh+GPGzKgsC7Fa6/VFJIENuGQTwN2fB57v\n0/bNPtv39fO8x4HHhzC+s6qyvIj/evsDahqOMTWUE+/hiIjEXNL/IjdS73r93arri0hiUuhHmDk+\nl6IxGfoyV0QSlkI/QriuX6STr4lIwlLo91FVHmJv4zFqtF5fRBKQQr+PymC9/urdOtoXkcSj0O9j\n1vg8CnPSVdcXkYSk0O8jJcVYWFbEaq3gEZEEpNDvR1V5iJqGY+xtPBbvoYiIxJRCvx+VZeH1+jrV\nsogkGoV+P2ZPzKMgW3V9EUk8Cv1+9NT1dfI1EUk0Cv1TqCoP8X5DK/tU1xeRBKLQP4XKsp71+irx\niEjiUOifwpxJ+eRnpemUDCKSUBT6p5CaYiwsC+nLXBFJKAr906gqL2LPoVb2N7XFeygiIjGh0D+N\nnvPrq64vIolCoX8acyblk5eVpqWbIpIwFPqnkZpiLJxepF/mikjCUOgPoLK8iF0Hj1LXrLq+iIx+\nCv0BfHjdXJV4RGT0U+gPYO6kfHIz01TiEZGEoNAfQFpqCgumF2q9vogkBIV+FCrLQ7xbf5S6FtX1\nRWR0U+hHoaeuv0Z1fREZ5RT6UTh/cj5jMlJV4hGRUS+q0DezJWZWbWY7zezefh7/oZltCG7bzawx\n4rHfmFmjmf1nLAd+NqWlplAxvUgnXxORUW/A0DezVOAB4FpgLnCTmc2N7OPud7v7Re5+EfBj4N8j\nHv574LOxG3J8VJWH2FF3hINHjsd7KCIigxbNkf5CYKe773L3dmAZcMNp+t8EPNmz4e6/BVqGNMoR\noLI8fH591fVFZDSLJvSnADUR27VB20nMbBpQBrx8JoMws9vNbJ2Zrauvrz+Tp54186YUkKO6voiM\ncrH+Incp8Iy7d53Jk9z9QXevcPeK4uLiGA8pNtJTU7hkWqHq+iIyqkUT+nuB0ojtkqCtP0uJKO0k\nmqryENUHWmg42h7voYiIDEo0ob8WmGlmZWaWQTjYl/ftZGazgUJgZWyHOHJU9db1VeIRkdFpwNB3\n907gTmAFsBV4yt03m9n9ZnZ9RNelwDJ398jnm9nrwNPAx82s1syuid3wz655U8aSnZ6q8+uLyKiV\nFk0nd38eeL5P2zf7bN93iudePtjBjTQZaeG6vr7MFZHRSr/IPUNV5UVs29/CYdX1RWQUUuifocqe\n8/DsUYlHREYfhf4ZuqCkgKz0FJV4RGRUUuifocy0VC6eWqgvc0VkVFLoD0JVeYht+5tpbFVdX0RG\nF4X+IFSVh3DXeXhEZPRR6A/ChaUFZKalsFqhLyKjjEJ/ED6s6+vLXBEZXRT6g1RZXsSWD5ppOtYR\n76GIiERNoT9IPXX9tSrxiMgootAfpItKx5KRlsJqnXxNREYRhf4gZaWnMr90rNbri8iootAfgsry\nEJv3NdHcprq+iIwOCv0hqCovotthnc7DIyKjhEJ/CC6eWkhGaoouoSgio4ZCfwiy0lO5qHSs1uuL\nyKih0B+iyvIi3t7bRIvq+iIyCij0h6iqPBSu6793ON5DEREZkEJ/iC6eWkh6qqnEIyKjgkJ/iLIz\nUrmwZKy+zBWRUUGhHwNV5SHe3tvEkeOd8R6KiMhpKfRjoLK8iK5uZ73q+iIywin0Y+CSaYWkpaiu\nLyIjn0I/BnIy0rigpIDVCn0RGeEU+jFSVR5iU20Tre2q64vIyBVV6JvZEjOrNrOdZnZvP4//0Mw2\nBLftZtYY8djNZrYjuN0cy8GPJJXlITpV1xeRES5toA5mlgo8AFwN1AJrzWy5u2/p6ePud0f0vwuY\nH9wvAr4FVAAOrA+em3DJWDGtkNSgrn/5zOJ4D0dEpF/RHOkvBHa6+y53bweWATecpv9NwJPB/WuA\nF929IQj6F4ElQxnwSDUmM415Uwq0Xl9ERrRoQn8KUBOxXRu0ncTMpgFlwMtn+txEUFUeYmNto+r6\nIjJixfqL3KXAM+7edSZPMrPbzWydma2rr6+P8ZDOnsryIjq6nDffaxy4s4hIHEQT+nuB0ojtkqCt\nP0v5sLQT9XPd/UF3r3D3iuLi0VsP76nr67q5IjJSRRP6a4GZZlZmZhmEg315305mNhsoBFZGNK8A\nFptZoZkVAouDtoSUl5XO+VMK9CMtERmxBgx9d+8E7iQc1luBp9x9s5ndb2bXR3RdCixzd494bgPw\nHcI7jrXA/UFbwqoqK2JDTSPbD7ToXDwiMuJYREaPCBUVFb5u3bp4D2PQXttez+ceXtO7nZORyoT8\nLMbnZTI+P4sJeZnh7fxMxudlMSE/vD0mc8DVsyIip2Rm6929YqB+SpoYu3zmOJ75wqXUHG6lrvk4\nB5qPc6Cljfrm42yqbeRAcxttHd0nPW9Mz86hz86gONhJ9Ow4tHMQkaFQgsSYmVExvYiK6UX9Pu7u\nNLd1Ut/SFt4hNLdR1xL823ycupY2NtSEdw7HO0/eOeRmpjE+P5MJeeEdxKk+ReRk6K0VkZMpGc4y\nM6MgO52C7HRmjM87Zb+enUNdc3jnUHfCTiK8g3jz/cMcaD5Oez87h7zMNCaPzWb2pDzmTspn7uR8\n5k7KJ5SbOZzTE5ERTqE/QkXuHGZOGGDncKyTAy1tvZ8WDgQ7hZqGVtbubuA/Nuzr7T8hPzNiJ1DA\n3Mn5TCvKISXFzsa0RCTOFPqjnJlRkJNOQU46s06xczh8tJ2tHzSz5YNmtuwL//vajoN0dYe/xB+T\nkcqciE8DcyfnM2tCHlnpqWdzKiJyFmj1TpJq6+hiZ92R3p1Az789y0xTU4xzisec9KmgaExGnEcu\nIv3R6h05raz0VM6fUsD5Uwp627q7ndrDx9jyQROb94V3BKt3N/DriPLQxPys3k8E500O7xBKC1Ue\nEhktFPrSKyXFmBrKYWoohyXnT+ptb+gpD0V8Kvjd9vre8lBuZhpzTvjCuICZE3JVHhIZgVTekUFp\n6+hix4EjbPmgqXdnsPWDlhPKQzOKc5k7OZ85k/I4d2I+syfmMT4vEzN9KhCJNZV3ZFhlpacyr6SA\neSUnlodqDree8Ilg5buHePatD8+xNzYnnXMn5DF7YnhHcO7EXGZNyCMvKz0e0xBJOgp9iZmUFGNa\naAzTQmO4dt6H5aHDR9upPtBC9f4Wtu1voXp/M796cy9Hjr/X22fK2OxgRxC+zZ6YT3nxGNJTdRln\nkVhS6MuwKxyTQVV5iKryUG+be/hL4+r9LVQf+HBn8Lvt9XQG3xWkpxrnFOdG7AjCnw4mF2SpRCQy\nSAp9iQszo7Qoh9KiHP5g7oTe9vbObt6tP3LCp4K+PzDLy0xjVuSOYEL4k0FBjkpEIgNR6MuIkpGW\nwpxJ+cyZlH9Ce9OxDnb0fiII3/5z4z6eWP3h6asn5mdFfCII32aMzyUzTauIRHoo9GVUKMhOP+lE\ndu7O/ua2E3YE2/a3sPLdQ7R3hc9HlJpilI0bw7kT8vjTihKuPHd8vKYgMiIo9GXUMjMmFWQzqSCb\nqyLCvKOrmz0Hj/buDLbtb2HtngZWbN7PTz5zCVdHlJNEko3W6UtSaGnr4DMPrWHrvmYeuqWCy2eO\n3msxi/Qn2nX6Wg8nSSEvK51HP7+A8uIx/Plj61izO6Gv2ilySgp9SRpjczJ4/LZKJo/N5tZH1rKx\npjHeQxI56xT6klTG5WbyxG1VFI5J53MPr2HrB83xHpLIWaXQl6QzsSCLJ26rIjs9lc8+tJp364/E\ne0giZ41CX5JSaVEOj99WiTt8+merqWlojfeQRM4Khb4krRnjc3n8tkqOdXTxqZ+vYn9TW7yHJDLs\nFPqS1OZMyufRWxdy+GgHn/75Kg4eOR7vIYkMK4W+JL2LSsfy0M0V7G08xmcfWkNTa0e8hyQybBT6\nIkBleYgHP1vBu3VHuPlf1/ReDEYk0UQV+ma2xMyqzWynmd17ij43mtkWM9tsZk9EtH/fzN4Jbp+M\n1cBFYu2js4r550/N5+29Tdz6yFqOtXfFe0giMTdg6JtZKvAAcC0wF7jJzOb26TMT+CqwyN3PA/4y\naP9D4GLgIqAS+IqZnXj6RJERZPF5E/nHGy9k7Z4G7nh8Pcc7FfySWKI50l8I7HT3Xe7eDiwDbujT\n58+BB9z9MIC71wXtc4HX3L3T3Y8Cm4AlsRm6yPC44aIpfP9PLuC17fXc9cRbdARn7BRJBNGE/hSg\nJmK7NmiLNAuYZWa/N7NVZtYT7BuBJWaWY2bjgKuA0r5/wMxuN7N1Zrauvr7+zGchEmM3Lijlvuvm\n8sKWA3zl6Y10dY+sExOKDFasTq2cBswErgRKgNfMbJ67v2BmC4D/AeqBlcBJn5fd/UHgQQifZTNG\nYxIZklsWldHa0cXf/aaa7PRU/vZP5ukyjTLqRRP6eznx6LwkaItUC6x29w5gt5ltJ7wTWOvufwP8\nDUDwBe/2IY9a5Cz54pUzaD3exT+/spPsjFS++UdzFfwyqkVT3lkLzDSzMjPLAJYCy/v0+TXho3yC\nMs4sYJeZpZpZKGi/ALgAeCFGYxc5K768eBa3LirjX3+/hx+8oGMWGd0GPNJ3904zuxNYAaQCD7v7\nZjO7H1jn7suDxxab2RbC5Zt73P2QmWUBrwdHRs3AZ9xdC6BlVDEzvvFHczjW0dl7xP8XV82I97BE\nBkVXzhKJUle38+WnNvDrDfv41nVz+fyisngPSaRXtFfO0jVyRaKUmmL8w59eyLGOLr793BZyMlL5\n5IKp8R6WyBnRaRhEzkBaagr/dNN8rphVzL3//jb/saHvmgaRkU2hL3KGMtNS+ZfPXMLC6UX81VMb\nWbF5f7yHJBI1hb7IIGRnpPLQLQuYN6WAu554i99t148KZXRQ6IsMUm5mGo9+fiEzxudyxy/WsXrX\noXgPSWRACn2RISjISecXf7aQksIcbn1kLRtqGuM9pFOqa27jqXU1fPHf1rPoey/z/d9s46hOIZ10\ntGRTJAb2N7Vx409X0tjazrLbL2Xu5PifTLar29lQ08ir1XW8Ul3HO3ubAZiQn8msCXm8vuMgE/Iz\n+eq1c7jhosn6pfEoF+2STYW+SIzUNLRy409X0t7ZzS/vuJQZ43PP+hgOH23ntR31vLytjte213O4\ntYMUg0umFXLlueO56tzxzJmUh5nx5vuHuW/5ZjbVNlExrZD7rj+P86cUnPUxS2wo9EXiYFf9EW78\n6SpSU+DpOy5jaihnWP9ed7ez5YNmXtkWPpp/q6YRdwiNyeCKc4u56tzxXD5zHGNzMk75/GfW1/L9\n32yjobWdpQum8pXFswjlZg7ruCX2FPoicbJtfzNLH1xFbmYaT3/hUiYVZMf09ZvbOnhjx0Fe2VbH\nq9vrqW8JX8z9wpKC8NH87PFcMKWAlJToyzVNxzr4p9/u4NH/2UNORip/dfUsPlM1jbRUfe03Wij0\nReJoU20jn/rZasbnZfLLOy6lOG/wR87uzo66I71H8+v2HKaz28nPSuOjs8JH8x+dVTykv9Fjx4EW\nvv3cFt7YeZBZE3K577rzuGzGuCG/rgw/hb5InK3d08DnHlrDtFAOy26vOmWJpT+t7Z38z85DvFJd\nx6vV9extPAbAnEn5XHVuMVfNHs/80rHDciTu7ryw5QDf+c8t1B4+xrXnT+Rrn5hDadHwlqpkaBT6\nIiPAGzsOcusja5kzKY/Hb6skLyv9lH33HDzKK9V1vFJdz6pdh2jv7CYnI5WPzBjHVbPHc+W5xTEv\nFZ1OW0cXP3ttFw+8uhN3+MIV5/CFK84hOyP1rI1BoqfQFxkhXtpygC88vp75U8fy6K0LyckIn+ew\nraOLNbsbeo/mdx88CsA5xWO4KqjNV0wvJDMtviG7r/EYf/vf23hu4z6mjM3m6384h2vPn6glniOM\nQl9kBHlu4z6+tOwtFs0YxzXnTeTV6jp+v/MQxzq6yExL4bJzQuGj+Vnjh33Fz2Ct3nWIby3fzLb9\nLVxaHuJb189l9sT4/x5BwhT6IiPM0+tquOeZTQCUFGbzsdnhdfNV5aFRUzLp7OrmybU1/OCFalra\nOvls1TTu/oNZFOScumwlZ4dCX2QEevP9w+RnpXNO8ZhRXR45fLSdf3xxO/+2+j0KstO555rZfHJB\nKalnsExUYkuhLyLDbsu+Zu57bjNrdjdw3uR8vn39eVRML4r3sJJStKGvX16IyKDNnZzPL2+v4sc3\nzafhaDv/519W8qVlb7G/qS3eQ5NTUOiLyJCYGdddOJnffvkK7vrYDP77nf187Aev8sArOzne2RXv\n4UkfCn0RiYmcjDS+vPhcfvtXV3D5zHH8/YpqFv/wNV7acoCRVkYeaTq7uqlpaGXHgZZh/1uq6YvI\nsHh9Rz3ffm4LO+uOcMWsYr553VzOKT77Zx4dCdyd+pbj1BxupabhGDUNrdQcbqX28DFqDreyr7GN\nrm5n/tSxPPvFRYP6G/oiV0TirqOrm8dWvsf/e3E7xzq6uPUjZdz1sRmn/WXyaOTuNB3rCId4EOg1\nDceCf8Phfryz+4TnjMvNpLQom9LCnN5/zxmfy4JBfhGu0BeREePgkeP8/W+qeWp9DaExmdx77Wz+\nZP6UMzoTaLy1tnd+GOoNrdT0BvwxahtaaelzFbL8rDRKi3I+DPWiHEoKw+FeUpgT899mKPRFZMTZ\nWNPIfc9t5q33G7mwdCz3LpnN5LFZpJiRkmKkGKSaYRbcTznxfooZFvTpuR+r3zu0d3azr/HYCUfp\nPSFfe7iVg0faT+iflZ5CSWEOpYXZJ4R7SWEOpUU5FGSf3U8zMQ19M1sC/AhIBX7u7t/rp8+NwH2A\nAxvd/VNB+98Bf0j4S+MXgS/5af6oQl8ksXV3O8++tZfv/WZb77UAhiLFCO80zEhJibhvBDuSiO2T\n2sP329q72N/cRndEMqWlGJPHZkeUYIIj9SDgx+VmjKgf2EUb+mlRvFAq8ABwNVALrDWz5e6+JaLP\nTOCrwCJ3P2xm44P2y4BFwAVB1zeAK4BXz2w6IpIoUlKM/31JCdecP5FXttXR3tlNlzvuTreHr+0b\neb/bHXfo8oj7QXu3h3civffdg+3gfnDr6iZ4zT73g36ZqSmUFGZTEnHEPjE/KyEvIjNg6AMLgZ3u\nvgvAzJYBNwBbIvr8OfCAux8GcPe6oN2BLCADMCAdOBCboYvIaJabmcZ1F06O9zCSTjS7sSlATcR2\nbdAWaRYwy8x+b2argnIQ7r4SeAX4ILitcPetQx+2iIgMRjRH+tG+zkzgSqAEeM3M5gHjgDlBG8CL\nZna5u7956iVeAAAEZ0lEQVQe+WQzux24HWDq1KkxGpKIiPQVzZH+XqA0YrskaItUCyx39w533w1s\nJ7wT+GNglbsfcfcjwH8Dl/b9A+7+oLtXuHtFcXHxYOYhIiJRiCb01wIzzazMzDKApcDyPn1+Tfgo\nHzMbR7jcswt4H7jCzNLMLJ3wl7gq74iIxMmAoe/uncCdwArCgf2Uu282s/vN7Pqg2wrgkJltIVzD\nv8fdDwHPAO8CbwMbCS/lfG4Y5iEiIlHQj7NERBKAzqcvIiInUeiLiCSREVfeMbN64L14j2MA44CD\n8R7EWaK5JibNNfFMc/cBlz+OuNAfDcxsXTS1s0SguSYmzTV5qbwjIpJEFPoiIklEoT84D8Z7AGeR\n5pqYNNckpZq+iEgS0ZG+iEgSUej3w8z2mNnbZrbBzNYFbUVm9qKZ7Qj+LQzazcz+ycx2mtkmM7s4\nvqMfmJk9bGZ1ZvZORNsZz8/Mbg767zCzm+Mxl4GcYq73mdne4P3dYGafiHjsq8Fcq83smoj2JUHb\nTjO792zPIxpmVmpmr5jZFjPbbGZfCtoT7r09zVwT8r2NKQ+uWKPbhzdgDzCuT9vfAfcG9+8Fvh/c\n/wThs4caUAWsjvf4o5jfR4GLgXcGOz+giPBJ9YqAwuB+YbznFuVc7wO+0k/fuYTPEZUJlBE+b1Rq\ncHsXKCd8QaCNwNx4z62f8U8CLg7u5xE+2+3cRHxvTzPXhHxvY3nTkX70bgAeDe4/CvyviPbHPGwV\nMNbMJsVjgNFy99eAhj7NZzq/a4AX3b3Bw1dMexFYMvyjPzOnmOup3AAsc/fjHj5F+E7CV47rvXqc\nu7cDPVePG1Hc/QN3fzO430L4BIlTSMD39jRzPZVR/d7GkkK/fw68YGbrgwu8AExw9w+C+/uBCcH9\naK4sNhqc6fxG+7zvDEoaD/eUO0iguZrZdGA+sJoEf2/7zBUS/L0dKoV+/z7i7hcD1wJ/YWYfjXzQ\nw58XE3bZU6LPD/gJcA5wEeHLeP4gvsOJLTPLBX4F/KW7N0c+lmjvbT9zTej3NhYU+v1w973Bv3XA\ns4Q/Ah7oKdsE//Zc/D2aK4uNBmc6v1E7b3c/4O5d7t4N/Izw+wsJMNfgYkW/Av7N3f89aE7I97a/\nuSbyexsrCv0+zGyMmeX13AcWA+8QvlpYzyqGm4H/CO4vBz4XrISoApoiPkqPJmc6vxXAYjMrDD5C\nLw7aRrw+37n8MeH3F8JzXWpmmWZWRviSn2uI7upxcWdmBjwEbHX3f4x4KOHe21PNNVHf25iK9zfJ\nI+1G+Fv8jcFtM/D1oD0E/BbYAbwEFAXtBjzAh1cIq4j3HKKY45OEP/p2EK5h/tlg5gfcSvgLsZ3A\n5+M9rzOY6y+CuWwi/D/4pIj+Xw/mWg1cG9H+CcIrRN7t+W9ipN2AjxAu3WwCNgS3TyTie3uauSbk\nexvLm36RKyKSRFTeERFJIgp9EZEkotAXEUkiCn0RkSSi0BcRSSIKfRGRJKLQFxFJIgp9EZEk8v8B\nrgYfcqBh4PIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x118ebead0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#retrieve train and validation summary object and read the loss data into ndarray's. \n",
    "val_mae  = np.array(val_summary.read_scalar(\"MAE\"))\n",
    "\n",
    "#plot the train and validation curves\n",
    "# each event data is a tuple in form of (iteration_count, value, timestamp)\n",
    "plt.plot(val_mae[:,0],val_mae[:,1],label='mae')\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* output the final MAE on validation dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-04T02:21:45.157466Z",
     "start_time": "2017-08-04T02:21:44.240825Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "creating: createMAE\n"
     ]
    }
   ],
   "source": [
    "res2 = ncf.evaluate(val_rdd,2800,[MAE()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-08-04T02:21:45.167142Z",
     "start_time": "2017-08-04T02:21:45.160480Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Evaluated result: 0.683162510395, total_num: 576, method: MAE'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'\\n'.join([str(r) for r in res2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "navigate_num": "#000000",
    "navigate_text": "#333333",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700",
    "sidebar_border": "#EEEEEE",
    "wrapper_background": "#FFFFFF"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "141px",
    "width": "251px"
   },
   "navigate_menu": true,
   "number_sections": false,
   "sideBar": true,
   "threshold": 4.0,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": true,
   "widenNotebook": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
